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A  blob  is  a  compact  region  lighter  (or  darker)  than  its 
background  surrounded  by  a  smoothly  curved  edge.  Blobs  can 
be  detected  using  cooperating  relaxation  processes  to  enhance 
their  interior  and  edge  probabilities.  A  discussion  of  the 
processes  working  independently  and  together  is  given.  The 
use  of  a  pyramidal  relaxation  structure,  the  use  of  the 
closedness  of  contours  as  an  additional  information  source, 
and  the  extension  of  cooperating  relaxation  processes  to  a 
time  sequence  of  images  are  also  discussed. 
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PREFACE 


Basic  to  most  image  understanding  tasks  is  the  segmenta¬ 
tion  of  objects  from  their  background.  In  simple  situations 
the  objects  to  be  extracted  may  be  "blobs".  A  blob  can  be 
characterized  as  a  compact  region  lighter  (or  darker)  than  its 
background  surrounded  by  a  smoothly  curved  edge.  In  principle, 
a  blob  can  be  extracted  by  thresholding  the  image  at  an  appro¬ 
priate  level,  or  its  bounds  can  be  defined  by  an  edge  detector. 
However,  because  of  noise  and  blur,  pure  object/backcrround  discri¬ 
mination  can  rarely  be  achieved  in  this  way.  This  thesis  poses 
a  fuzzy  classification  scheme  for  blob  detection.  It  incor¬ 
porates  ideas  used  in  non-probabilistic  methods  making  use  of 
edge/border  coincidence,  and  implements  these  ideas  in  the  form 
of  a  probabilistic  relaxation  process. 

In  Chapter  I  probabilistic  relaxation  is  discussed  and  blob 
detection  using  edge/border  coincidence  is  described. 

In  Chapter  II  the  behavior  of  the  edge  process  and  the  gray 
level  process  are  examined,  first  when  operating  separately  and 
then  when  interacting  in  a  joint  relaxation  process. 

Chapter  III  discusses  several  extensions  to  the  blob  ex¬ 
traction  processes.  These  include  a  pyramid  structure  for 
multilevel  relaxation,  the  use  of  curvature  information  as  a 
basis  for  the  initial  estimation  of  "inside"  and  "outside"  class 
probabilities,  and  a  scheme  for  computing  compatibility  coef¬ 
ficients  for  detecting  objects  in  a  time  sequence  of  one¬ 
dimensional  images. 


CHAPTER  I 


BACKGROUND 

1. 1  Probabilistic  Relaxation 

A  primary  step  in  the  analysis  of  an  image  is  the  discri¬ 
mination  or  classification  of  the  parts  of  the  image.  A  low 
level  symbolic  description  of  the  image  based  on  local  evidence 
(i.e.,  gray  level,  gradient  magnitude,  curvature)  can  be 
built.  However,  local  evidence  may  be  misleading,  causing 
ambiguities  in  the  classification.  Ambiguities  arise  from  the 
sensitivity  of  local  evidence  to  noise,  or  multiple  responses 
to  the  same  gray  level  pattern,  as  in  computing  slope  infor¬ 
mation.  The  ambiguity  problem  is  especially  severe  if  the 
classification  is  done  in  parallel  with  each  point  being  clas¬ 
sified  without  reference  to  any  classification  decisions  that 
may  have  already  been  made  at  other  points.  To  disambiguate  the 
classification,  an  iterative  process  which  makes  use  of  the 
relationships  among  the  classes  to  reduce  or  eliminate  the 
ambiguity  is  employed.  This  computational  process  is  called 
"relaxation  labeling"  [1] . 

The  relaxation  labeling  method  uses  probabilistic  classi¬ 
fication  rather  than  making  firm  classification  decisions 
immediately.  Classification  probabilities  at  a  given  iteration 
are  allowed  to  depend  on  decisions  made  at  the  previous  iteration. 
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The  probability  updating  at  a  given  iteration  is  done  in 
parallel  over  the  entire  image. 

To  convey  the  general  idea  of  the  method,  a  simplified 
overview  is  given  in  [2] .  We  initially  assign  to  each  point 
P  the  probability  p^  that  it  belongs  to  each  of  its  possible 
classes.  We  then  examine  the  probabilities  at  neighboring 
points,  increasing  p^  if  supporting  evidence  is  found  for  it 
(i.e.,  if  there  exist  high  probabilities  that  the  neighbors 
of  P  have  compatible  classifications) ,  or  decrease  it  if  con¬ 
tradictory  evidence  is  found  (i.e.,  there  exist  high  probabi¬ 
lities  of  incompatible  classification  at  P's  neighbors).  The 
updating  is  done  in  parallel  for  every  P  and  every  p^,  and  the 
process  can  be  iterated  as  many  times  as  desired. 

Two  formal  models  of  the  process  have  been  formulated  and 
applied  [3]  [4] .  We  now  briefly  describe  the  models.  Let 
A  =  {a^,...,an>  be  the  set  of  objects  (possibly  single  points) 
in  a  scene,  each  of  which  belongs  to  one  of  the  classes  of  the 


set  A  =  {Xlf...,X  }.  With  each  a^  we  associate  a  probability 

vector  (p^^, . . .  ,p^m)  ,  where  p^  is  an  estimate  of  the  probability 

that  a.  belongs  to  the  class  X  ,  where  E  p. (X)  =  1,  and 
1  *  X=A  1 

0  *  p^(X)  *  1  for  all  a^  €  a.  This  estimate  is  based  on  some 

conventional  analysis  of  a^.  For  example,  the  initial  esti¬ 
mate  of  the  edgeness  of  the  point  a^  may  be  proportional  to 
the  magnitude  of  the  gradient  of  the  image  at  a^.  For  each 


pair  of  neighboring  objects  a^,a 


j 


(given  a  specified  neighbor 


3 


relation)  and  each  pair  of  classes  X^,X^,  there  is  a  measure 
of  compatibility  between  object  a^  belonging  to  class  X^  and 
object  a^  belonging  to  class  X^.  Compatibilities  are  given 
by  the  function  r^j(X,X')  where  -1  *  r  &  +1  and 

a)  if  X  at  a ^  is  compatible  with  X'  at  a ^  ,  then 
r..(X,X')  >  0; 

b)  if  X  at  a^  is  incompatible  with  X'  at  a^.-  then 
ri  j (X, X 1 )  <  0; 

c)  if  neither  labeling  is  constrained  by  the  other  ("don't 
care"  situation),  then  r^j(X,X’)  =  0; 

d)  the  magnitude  of  r^j  represents  the  strength  of  the 
compatibility. 

Compatibility  coefficients  can  be  computed  as  correlation 
functions,  by  a  priori  evaluation  of  the  ways  in  which  the 
classes  can  interact,  or  based  on  mutual  information  of  the 
labels  at  neighboring  points  (i.e.,  if  two  labels  have  a  high 
positive  correlation,  it  is  expected  that  they  will  have  a 
high  mutual  information,  and  vice  versa) .  The  pairwise  compati¬ 
bilities  among  the  labels  at  adjacent  points  computed  by  mutual 
information  are  estimated  using  ratios  of  probabilities  given  by 
p..j(X,X') 

pTxTpTFT 

where  i,j  specifies  a  neighborhood  direction  and  X,X'  range 
over  the  label  set.  For  a  detailed  discussion  of  computing 
compatibility  coefficients  by  mutual  information  see  [5]. 


4 


The  new  estimate  of  the  probability  of  X  at  a^  at  the 
(k+l)st  iteration  is  a  function  of  both  the  previous  estimate 

If 

for  that  probability,  p^(A),  and  the  contribution  from  the 
probability  distributions  on  the  neighboring  label  sets, 
q£(X),  VA€A,  given  by  [3] 


Pf+1)  (X) 


pJ(A) [ 1+q ^ ( X ) ] 

^p.(a)  [l+qj(a)] 
a  1  1 


where 


q^(X)  =  E  Er .  .  (A,a)p  .  (a) 
j  a  J  -> 

This  updating  scheme  is  useful  when  compatibilities  are 
expressed  as  positive  and  negative  coefficients.  When  com¬ 
patibilities  are  computed  by  mutual  information  a  non-negative 
coefficient  scale  results,  where  0  represents  high  incompati¬ 
bility  and  high  values  represent  high  compatibilities.  The 
second  model  works  with  these  compatibilities.  The  new  estimate 
of  the  probability  of  X  at  a^  at  the  (k+l)st  iteration  is 
given  by  [4] 


p(k+l)  w  .  pj[(MSr11O..Qp>) 

1  Z  Ir..  (A,a)p^(A)p^(a) 

A  a  ^  1  3 

When  the  a's  at  a^  are  highly  compatible  with  p i  ( X )  the  sum 
will  be  high,  low  otherwise,  and  p^(A)  will  increase  relative 
to  other  p's. 

Both  iteration  schemes  yield  comparable  results.  A  study 
of  the  convergence  properties  of  relaxation  processes  is  found 


in  16]  [7).  Relaxation  methods  have  been  used  in  breaking 
substitution  ciphers  [8],  handwriting  segmentation  [9]  [19], 
line  and  curve  enhancement  [1] [10] [11]  and  edge  enhancement 
[12] [13],  among  other  applications. 
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1. 2  Blob  Detection 

Segmentation  of  objects  from  their  background  is  basic  to 
many  image  understanding  tasks.  This  paper  deals  with  the 
relatively  simple  case  in  which  the  object  is  a  "blob".  A 
blob  can  be  characterized  as  a  compact  region  lighter  (or 
darker)  than  its  background  surrounded  by  a  smoothly  curved 
edge.  Not  all  images  conform  to  this  rudimentary  model; 
for  example,  consider  images  of  clouds  whose  regions  lack  well 
defined  outlines,  or  images  that  consist  of  subparts  with 
differing  textures.  However,  the  model  is  applicable  to  many 
imaging  tasks  including  thermal  imagery  analysis,  chromosome 
classification,  and  industrial  automation. 

In  principle,  blobs  can  be  extracted  by  thresholding  the 
image  at  an  appropriate  level.  However,  if  the  image  is  noisy, 
thresholding  will  produce  noisy  results  which  may  not  be  cleaned 
up  in  postprocessing.  Thresholding  may  extract  regions  that 
are  not  bounded  by  edges  but  are  smooth  continuations  of  the 
background  if  the  gray  level  fluctuations  in  the  background 
cross  the  threshold  level.  Edge  detection  is  also  sometimes 
useful  in  object  extraction.  However,  the  edge  detector  may 
respond  in  the  interior  of  the  object  or  background  as  a  result  of 
noise  or  may  fail  to  respond  strongly  on  the  object/background 
border  because  of  blur. 

Milgram  [14] [15]  has  investigated  the  use  of  edge/border 
coincidence  as  a  method  for  object  detection  using  the  "Superslice" 


algorithm.  The  basic  concept  is  the  matching  of  border  points 
of  connected  components  with  corresponding  edge  values.  In 
this  approach,  an  edge  detector  is  run  over  the  image  and  the 
response  thinned  to  an  "edge  map".  The  image  is  then  thresholded 
and  connected  components  of  above-threshold  points  are  extracted 
(see  Rosenfeld  and  Kak  [16]  for  a  discussion  of  connected 
components  and  edge  detectors) .  The  components  are  accepted 
as  objects  or  rejected  as  noise  based  on  the  coincidence  of 
the  edge  map  with  the  region  boundary.  This  process  is  carried 
out  at  various  thresholds  and  at  each  threshold  the  surviving 
regions  are  compared  with  the  survivors  of  earlier  thresholds. 
Only  those  regions  that  best  match  the  edge  map  are  used  to 
describe  the  actual  objects  in  the  image. 

Nakagawa  used  edge/border  coincidence  as  an  aid  in  edge 
extraction  [17] .  He  was  able  to  reduce  the  noisiness  in  the 
edge  detector  response  by  selecting  those  edge  points  that  lie 
on  region  borders  in  the  thresholded  image.  The  selected  edge 
points  were  linked  into  continuous  curves  by  following  the 
region  borders. 

Parikh  experimented  with  the  use  of  border  edge  strength 
values  to  determine  a  threshold  for  cloud- type  objects  [18] . 

Edge  strength  was  defined  to  be  the  sum  of  the  edge  values 
for  all  border  points  of  a  given  connected  component  divided 
by  the  number  of  border  points.  The  border  edge  strength 
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feature  failed  to  segment  the  cloud-type  objects  because  of  the 
ill-defined  nature  of  the  borders. 

Relaxation  applied  to  the  detection  of  edges  was  success¬ 
fully  investigated  by  Schachter  et  al.  [12].  In  their 
approach,  the  gradient  magnitude  was  computed  by 
MAG  =/(AxF)2  +  (AyF)2 

where  F  is  the  image  and  A^F,  AyF  are  the  x,y  components  of 
the  gradient  respectively.  The  edge  direction  was  given  by 


0 


tan 


The  probability  of  an  edge  at  a  point  (x,y)  was  defined  by 


p(x,y) 


MAG ( x ,  y ) 
max  MAG ( u , v ) 
u,v 


where  the  max  was  taken  over  the  entire  image.  P(x,y)  = 

1  -  P(x,y)  defined  the  no-edge  probability.  Edge/edge  com¬ 
patibility  coefficients  were  computed  based  on  smoothness  of 
slope  continuation  for  edge/edge;  edge/no  edge  and  no  edge/ 
no  edge  interactions  were  also  defined.  In  brief,  no  edge 
reinforces  no  edge;  edge  reinforces  edge  if  they  smoothly 
continue  one  another;  and  edge  reinforces  no  edge  if  they  are 
alongside  one  another. 

To  apply  relaxation  to  thresholding  [21],  "light"  and  "dark" 
probabilities  are  initially  assigned  to  image  points  based  on 
tbair  gray  levels.  The  probabilities  are  iteratively  adjusted 
at  each  point  based  on  the  probabilities  at  the  neighboring 


i 
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points,  i.e.,  light  reinforces  light  and  dark  reinforces 
dark.  This  has  the  effect  of  shifting  the  probabilities  ini¬ 
tially  assigned  to  noise  points  so  as  to  make  them  more  con¬ 
sistent  with  their  surroundings.  After  a  number  of  iterations, 
light  probabilities  should  become  uniformly  high,  and  vice 
versa,  producing  a  bimodal  histogram  with  peaks  at  opposite 
ends  of  the  gray  scale,  making  threshold  selection  easier  and 
resulting  in  a  non-noisy  binary  image.  However,  the  process 
still  may  extract  regions  not  bounded  by  edges. 

This  thesis  uses  the  ideas  of  relaxation  and  the  ideas  of 
edge/border  coincidence  in  a  probabilistic  segmentation  scheme 
for  blob  detection.  Superslice,  a  non-probabilistic  scheme, 
made  two  independent  decisions  based  on  gray  level  (thresholding) 
and  edge  strength  (selection  of  maxima) ,  then  checked  for  coin¬ 
cidence  to  determine  if  an  object  was  detected.  The  joint 
relaxation  process  described  in  this  paper,  on  the  other  hand, 
never  makes  decisions.  It  estimates  the  probabilities  based 
on  gray  levels  and  edge  strengths,  and  then  iteratively  adjusts 
these  probabilities  so  that  both  types  of  information  are  able 
to  interact.  This  process  thus  incorporates  the  principle  of 
convergent  evidence  in  a  deferred  commitment  scheme. 


CHAPTER  II 


BLOB  DETECTION 

In  this  chapter  we  examine  the  edge  and  gray  level  pro¬ 
cesses  operating  separately,  and  then  the  joint  process  in 
which  they  are  both  combined. 

2 . 1  Edge  Relaxation 

We  compute  the  initial  edge  probabilities  as  follows. 

Let  e^ (i  =  1,...,8)  be  a  measure  of  the  gray  level  difference 
at  point  P  in  direction  45i°,  where  e^  is  computed  by  apply¬ 
ing  these  masks,  first  proposed  by  Prewitt,  at  every  picture 
point: 

-101  011  111  110 

-1P1  -1P1  OPO  IP-1 

-10  1  -1-10  -1  -1  -1  0  -1  -1 

We  take  the  absolute  value  as  the  edge  magnitude  and  let  the 
sign  indicate  the  edge  orientation  (i.e.,  a  90°  edge  is  a 
negative  270°  edge) .  The  operator  has  the  effect  of  thickening 
edges,  i.e.,  producing  edge  responses  on  the  leading  and  trail¬ 
ing  sides  of  the  object  border.  To  obtain  usable  edge  infor¬ 
mation,  the  output  must  be  thinned  by  suppression  of  non-maxima 
of  the  edge  magnitude  in  the  direction  across  the  edge  at  each 
point. 

Figure  1  shows  the  FLIR  image  of  a  tank  displayed  as  gray 
levels.  Figure  2a  shows  the  edge  detector  output  with  non- 
maximal  responses  suppressed.  This  illustrates  the  problem  of 
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the  edge  detector  responding  to  varying  intensity  patterns 
in  the  background.  A  histogram  of  the  edge  strengths  is 
shown  in  Figure  2b.* 

Let  E  be  the  largest  e  value  at  any  point  of  the  image, 

and  let  ep  be  the  largest  value  at  point  P.  Pe  =  ep/E  is 

taken  to  be  an  estimate  of  the  edge  probability  at  P  and 

Pn  =  l-p_  as  an  estimate  of  the  no  edge  probability.  Further, 

8 

p.  =  (e./s)p  ,  where  s  =  Z  e. ,  is  taken  as  an  estimate  of  the 
1  1  e  i=l  1  8 

probability  of  an  edge  in  direction  45i°  at  P.  Thus,  Z  p.  =  p 

i=l  1  e 

and  pe  +  pn  =  1.0.  Figure  3a  shows  the  initial  edge  classi¬ 
fications  with  pixels  having  maximum  edge  probability  (regard¬ 
less  of  direction)  rescaled  as  gray  levels  between  0  and  63, 
and  pixels  having  maximum  no  edge  probabilities  displayed  as 
black. 

The  pairwise  compatibilities  among  p, ,...,pQ,p  at  adja- 
cent  points  are  estimated  using  mutual  information  [5] .  A  few  of 
the  coefficients  are  shown  in  Figure  4,  showing  that  the  rela¬ 
tive  strengths  of  the  coefficients  support  the  intuitive  con¬ 
straints  of  no  edge  reinforcing  no  edge,  edge  reinforcing  edge 
if  they  smoothly  continue  one  another,  and  edge  reinforcing 
no  edge  (and  vice  versa)  if  they  are  alongside  one  another. 

This  has  the  effect  of  strengthening  the  appropriate  edge 
probabilities  at  points  that  lie  along  smooth  edges,  and 

*  The  scale  in  all  histogram  pictures  indicates  the  number 
of  pixels  per  displayed  dot. 


strengthening  the  no  edge  probability  elsewhere.  These 
coefficients  are  used  in  the  relaxation  process  of  [4]  . 

Figure  3(b-i)  shows  eight  iterations  of  the  edge  relax¬ 
ation  process  applied  to  Figure  3a.  Figure  5(a-i)  are  the 
histograms  of  the  respective  iterations.  As  the  relaxation 
process  iterates  the  edge  responses  clustered  at  the  lower 
end  of  the  scale  in  Figure  2b  migrate  in  two  directions. 

Those  points  which  have  low  edge  support  have  increased  no 
edge  probabilities  and  become  zero  while  those  points  receiving 
support  have  increased  edge  probabilities  and  move  toward  the 
higher  values.  The  no  edge  points  show  up  as  a  spike  at  gray 
level  zero.  The  blob  edge  becomes  significantly  enhanced  with 
the  blob  being  completely  defined  after  four  iterations.  As 
the  process  continues  some  of  the  edges  in  the  background 
begin  to  come  out;  however,  the  improvement  over  Figure  2a 


is  obvious. 
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2 . 2  Light/dark  Relaxation 

The  light/dark  process  was  initially  designed  as  an 
automated  thresholding  process  wherein  the  gray  levels  would 
decide  whether  they  belonged  to  the  light  class  or  dark  class  . 

The  result  should  be  a  bimodal  histogram,  which  makes  threshold 
selection  obvious.  (See  [21]  for  a  discussion  of  thresholding 
using  light/dark  relaxation.)  In  the  image  domain  with  which 
we  are  experimenting,  involving  homogeneous  objects  on  a  con¬ 
trasting  background,  the  light  class  can  be  equated  with  blob 
interior  and  the  dark  class  with  blob  exterior.  As  it  relates 
to  edge/border  coincidence,  this  process  has  the  joint  role 
of  supplying  the  border  information  and  classifying  the  object 
interior.  The  border  is  defined  by  interior  points  on  the 
light  sides  of  edges.  However,  we  first  examine  the  behavior 
of  the  process  without  the  edge  interaction. 

We  compute  the  initial  light/dark  probabilities  as 
follows.  Let  g  be  the  gray  level  of  point  P,  and  let  b,w  be 
the  lowest  and  highest  gray  levels  in  the  image,  so  that 
b  *  g  *  w  for  all  P.  p  =  is  taken  as  the  estimate  of 

the  probability  that  P  is  white,  and  p^  =  as  the  probability 

that  P  is  black.  Figure  6a  shows  che  initial  light/dark  clas¬ 
sifications  of  Figure  1  redisplayed  as  gray  levels  where 

g  =  b+p  (w-b)  =  w-p.(w-b).  Because  the  initial  probabilities 
W  D 

are  based  on  gray  levels  and  then  redisplayed  as  gray  levels. 
Figure  6a  is  a  copy  of  the  original  image. 


The  compatibilities  between  the  black  and  white  classes 
at  adjacent  points  are  estimated  using  ratios  of  average 
probabilities,  as  in  the  previous  section.  Figure  7  shows 
a  few  of  these  compatibilities;  we  see  that  interior  reinforces 
interior,  exterior  reinforces  exterior,  and  adjacent  interior/ 
exterior  classes  are  contradictory. 

Figure  6{b-i)  shows  eight  iterations  of  the  relaxation 
process  applied  to  Figure  6a.  Displayed  in  this  figure  is 
the  strength  of  each  point's  light  classification.  This  is 
interpreted  as  the  probability  that  each  point  belongs  to  the 
interior  class.  As  the  process  iterates  the  blob  interior 
grows.  After  three  iterations  those  ambiguously  classed  points 
on  the  blurred  edge  become  classified  as  interior.  However, 
with  no  inhibitory  process  to  temper  this  growth,  background 
points  neighboring  the  blob  interior  begin  to  accept  the  in¬ 
terior  support  and  become  classified  as  interior  themselves. 
Some  areas  in  the  background  which  do  not  look  blob-like 
(lacking  smoothly  curving  edges)  have  also  been  somewhat  en¬ 
hanced.  This  is  due  to  their  relatively  high  gray  level  and 
the  fact  that  supporting  evidence  is  based  only  on  gray  level. 

Figure  8(a-i)  show  the  histograms  of  the  corresponding 
iterations  shown  in  Figure  6 (a-i) .  The  process  produces  a 
discrimination  between  the  blob  and  the  background  evident 
by  the  widening  valley  between  the  exterior  class,  peaked  at 


the  low  end  of  the  histogram  around  gray  level  seven,  and  the 
interior  class  peak  gaining  strength  around  gray  level  43.  A 
difference  between  the  objective  of  this  process  and  the 
thresholding  process  is  that  this  produces  no  segmentation 
since  no  actual  thresholding  has  been  done. 


2.3  Joint  Relaxation 


We  now  allow  both  information  sources  to  interact  in  a 
multi-process  relaxation  scheme.  Given  the  two  initial  sets 
of  probabilities  for  each  point,  compatibility  coefficients 
can  be  defined  between  p^,  pw,  p^,...,Pg,  Pn  in  the  same 
manner  as  in  the  previous  sections.  The  edge/border  inter¬ 
action  is  embodied  in  the  inter-process  compatibilities  where 
edge  reinforces  light  if  it  is  on  the  light  side  of  the  edge 
(and  vice  versa),  edge  reinforces  dark  if  it  is  on  the  dark 
side  of  the  edge  (and  vice  versa),  and  the  no  edge/ (light, dark) 
interations  are  "no  information"  situations.  The  intra-process 
compatibilities  preserve  the  same  reinforcement  relations  as 
previously  detailed.  Figure  9  is  a  sampling  of  these  coeffi¬ 
cients. 


We  compute  adjusted  probabilities  using  the  relaxation 
formula  by  allowing  the  classes  to  interact  as  before  but  nor¬ 
malizing  the  two  sets  separately.  At  each  iteration  the  up¬ 


dated  light/dark  estimates  are  normalized  by 


pA  +  p™ 

b  w 


where  t  =  b  or  w  and  p£  is  the  updated  unnormalized  estimate 
of  L,  and  the  updated  edge  estimates  by 


t . . .  ^ 


p8  +  pA 
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Figure  10(a-i)  shows  the  results  of  the  initial  classi¬ 
fication  and  eight  iterations  of  the  joint  relaxation  process 
applied  to  Figures  3a  and  6a.  The  figure  consists  of  a  pair 
of  images  for  each  iteration.  In  one  pw  is  displayed  and  in 
the  other  pe  is  displayed.  Allowing  both  sources  of  infor¬ 
mation  to  interact  has  sped  up  the  blob  edge  definition. 

Operating  in  isolation,  as  in  Figure  3,  four  iterations  were 
needed  to  define  the  blob  completely.  Operating  jointly,  only 
three  iterations  are  needed.  The  edges  which  come  out  of  the 
background  in  the  single  process  are  greatly  inhibited  in  the 
joint  process.  A  comparison  of  Figures  2a,  3i  and  lOi  (edge) 
shews  a  dramatic  improvement.  One  shortcoming  of  the  light/dark 
process  operating  alone  was  the  blob's  growth  into  the  back¬ 
ground.  As  a  result  of  interacting  with  the  edge  information 
this  expansion  is  contained,  yet  not  entirely  inhibited  outside 
of  the  defined  edge.  The  containment  is  most  observable  in 
the  latter  iterations  of  the  process.  The  emergence  of  white 
patches  from  the  background  not  bounded  by  edges  is  unimproved 
by  the  increased  information.  This  is  because,  as  can  be  seen 
from  the  edge  process  alone,  there  are  edges  in  the  background 
giving  support  to  the  light  class  on  the  light  side  of  the 
edge,  and  no-edge  probabilities  have  no  inhibitory  effect  on 
either  the  light  or  dark  probabilities. 

The  histograms  of  Figure  10  (a-i)  are  shown  in  Figure  ll(a-i). 
The  effect  of  the  joint  process  on  the  light/dark  classifications 


can  be  analytically  supported  by  a  comparison  of  the  light/ 
dark  process  histograms  of  this  figure  with  the  histograms 
of  the  light/dark  process  operating  alone  (Figure  8) .  The 
light/dark  peaks  are  separated  by  a  much  sharper  valley  in 
the  joint  process,  meaning  that  fewer  points  are  ambiguously 
classified  (relatively  equal  light/dark  probabilities)  and  thus 
there  is  greater  discrimination  between  the  object  and  the 
background.  The  effect  of  the  joint  process  on  the  edge/no 
edge  classifications  can  also  be  analyzed  in  the  light  of  a 
comparison  of  the  edge  process  histograms  of  this  figure  and 
the  histograms  of  the  edge  process  operating  alone  (Figure  5) . 
Although  the  strongest  edge  in  the  joint  process  is  weaker 
than  in  the  single  process,  the  bulk  of  the  edges  are  stronger, 
and  the  ambiguously  classified  edges,  appearing  around  the 
low  middle  gray  levels  in  the  single  process  histogram,  have 
been  eliminated.  This  reflects  the  inhibition  of  edges  in  the 
background  as  seen  in  the  joint  gray  level  display. 

A  modification  to  the  joint  relaxation  process  was  tried 
in  order  to  deal  with  the  problem  of  the  blob's  growth  into 
the  background,  p^  and  pw  are  initialized  based  on  a  "borderness 
measure,  where  Pw  is  initially  high  only  adjacent  to  the  light 
sides  of  edges  and  low  elsewhere.  (The  idea  that  the  human 
visual  system  "colors  in"  regions  based  on  gray  levels  adjacent 
to  their  edges  is  well  known' to  perception  psychologists  [22].) 
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Borderness  values  are  computed  by  adding  the  difference  value 
e^  (the  result  of  applying  each  mask)  to  the  points  marked 
by  l's  if  e^  >0,  or  the  absolute  value  of  e^  to  the  points 
marked  by  -l's  if  e.  <  0,  in  each  of  the  masks  shown  in  the 
previous  section.  The  result  is  a  set  of  "borderness"  values 
which  are  high  on  the  light  sides  of  edges  and  low  elsewhere. 

p,  and  p  .  are  initialized  based  on  a  combination  of  the 
b  w 

gray  level  and  the  borderness  values  at  each  point.  Let  B 

be  the  maximum  borderness  value  in  the  image.  Let  3  be  the 

borderness  value  at  point  P,  and  let  PQ  =  3/B.  Let  p*  = 

p  w 

apw  +  (l-a)pg,  where  0  *  a  1  and  pw  is  computed  as  before, 
and  let  p*  =  1-p*  . 

With  the  edge  initialization,  compatibility  coefficients 
and  the  relaxation  process  carried  out  as  in  the  joint  process. 
Figure  12(a-c)  show  the  results  of  the  initial  classification 
and  eight  iterations  with  a =  .5.  This  is  an  obvious  improve¬ 
ment  over  the  results  in  Figure  10.  The  object  definition  is 
sharper  and  the  emergence  of  light  patches  from  the  background 
has  been  effectively  inhibited.  Improvement  has  also  been  made 
in  the  edge  classifications.  The  histograms  of  this  process. 
Figure  13(a~i),  support  what  is  observable  in  the  gray  level 
display. 
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CHAPTER  III 


EXTENSIONS 


This  chapter  discusses  several  extensions  of  the  blob 
extraction  process. 

3 . 1  Pyramid  Structure 

The  one-level  relaxation  structures  of  the  previous 
chapter  can  be  extended  to  a  multi-level  hierarchy  where 
each  successively  higher  level  in  the  "pyramid"  is  a  lower 
resolution  image  of  the  base  image.  At  an  appropriately  re¬ 
solved  level  blobs  look  like  local  spots,  enabling  immediate 
interaction  between  the  blob  interior  and  the  edges  on  all 
sides.  In  the  one-level  processes  a  considerable  number  of 
iterations  are  required  in  order  for  the  parts  of  a  large 
blob  to  reinforce  one  another.  In  the  pyramid  structure,  in¬ 
formation  passes  up  and  down  in  the  pyramid  (inter-level  inter¬ 
actions  and  intra-level  interactions)  facilitating  faster  pro¬ 
pagation  of  information. 

To  create  the  gray  level  pyramid,  on  which  initial  class 
estimations  are  based,  a  base  image  is  successively  block 
averaged  (using  a  4-pixel  block  size) ,  creating  the  lower 
resolution  levels.  Let  L^,L2 , • . . ,Ln  be  n  levels  in  a  pyramid 
where  Ln  is  a  higher  resolution  level  than  Ln+^»  and  let  PL 

n 

be  a  pixel  on  level  n.  In  this  structure  PL  is  adjacent 

n 

to  13  pixels:  a  father  in  Ln+^/  8  brothers  in  Ln,  and  4  sons  in 
L  P 

n-l*  L  is  the  father  of  PT  (and  PT  is  the  son  of  PT  ) 
n+1  Ln  n  Ln+1 
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if  PT  was  one  of  the  4  points  averaged  to  compute  PT  's 
n  Ln+1 

value.  The  8  brothers  are  the  8  adjacent  points  of  the 

one-level  process. 

Probabilities  are  updated  by  the  same  method  as  in  the 

one  level  process,  based  on  class  interactions  between  adjacent 

points,  but  with  the  information  from  the  three  interacting 

levels  weighted  differently,  i.e. 

PL  =  aF  +  8B  +  yS 
n 

where  F  is  the  father  in  L  , , 

n+1 

B  =  1  P,.  ..  ,  the  sum  of  the  updated  estimates  of 

J 

the  eight  brothers 

o  =  IP 

.  o  (k,£)T  ,the  sum  of  the  updated  estimates  of 
Ln-1 

the  four  sons 

and  a  +  8  +  y  =  1. 

Constrained  by  computer  memory  limitations,  a  three-level 
pyramid  (32x32,  16x16,  8x8)  was  created  based  on  the  image  in 
Figure  1;  it  is  shown  in  Figure  14.  The  pixels  were  initially 
classified  light/dark  by  the  process  described  in  Chapter  II 
and  displayed  in  Figure  15a  by  the  same  scheme.  The  compati¬ 
bility  coefficients  were  computed  by  mutual  information. 

Figure  15(b-i)  shows  eight  iterations  of  the  light/dark 
pyramid  process  applied  to  Figure  15a  with  a  =  8  =  y  =  .33. 

We  would  expect  multi-level  processes  to  be  more  informed  than 
single  level  processes,  converging  faster  with  more  discriminative 


results;  however  this  was  not  the  case.  The  multi-level 
process  performed  identically  to  the  single-level  process. 
Based  on  the  rationale  that  the  sons  are  less  informed  than 


their  fathers,  it  seems  reasonable  that  their  evidence  should 
not  be  allowed  to  reduce  the  class  estimation.  Hence  weights 
of  a  =  .25,  8  =  .75 ,  and  y  =  0.  were  used,  but  the  results 
were  still  the  same. 

Since  this  was  just  an  initial  step  in  the  investigation 
of  pyramidal  structures  for  relaxation,  the  default  option 
of  computing  compatibility  coefficients  by  mutual  information 
was  used.  A  more  intelligent  weighting  scheme  might  have 
produced  better  results.  Other  experiments  are  currently 
being  conducted  with  multi-level  processes  where  the  higher 
levels  determine  the  lower  level  compatibilities.  Work  on 
pyramidal  structures  for  object  extraction  will  be  continued 
in  the  future,  but  is  outside  the  scope  of  this  thesis. 


1 
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3 . 2  Curvature 

An  additional  information  source  investigated  was  the 
use  of  curvature  as  a  basis  for  initial  estimation  of  "inside" 
and  "outside"  class  probabilities  to  augment  the  "light"  and 
"dark"  classes  in  object  interior/exterior  interpretation. 
Initially,  most  points  have  equal  probability  of  being  inside 
or  outside  (assuming  we  do  not  know  whether  the  blobs  are  light 
and  the  background  dark  or  vice  versa) ,  except  that  points  ad¬ 
jacent  to  an  edge  on  the  side  atoay  from  the  center  of  curvature 
have  higher  probability  of  being  outside  while  those  points  on 
the  side  toward  the  center  of  curvature  have  higher  probability 
of  being  inside.  These  probabilities  then  reinforce  one  an¬ 
other;  inside  reinforces  inside  and  outside  reinforces  out¬ 
side  for  neighboring  pairs  of  points  to  the  extent  that  they 
are  not  separated  by  an  edge.  Initially,  the  probabilities 
at  a  given  point  adjacent  to  the  blob  border  will  depend  on 
whether  the  border  is  convex  or  concave  at  that  point.  Even¬ 
tually,  the  interior  of  the  blob  should  be  uniformly  labeled 
"inside"  and  the  exterior  labeled  "outside"  with  high  probabi¬ 
lity.  This  method  can  be  used  to  label  the  interior  of  a 
closed  curve  even  if  it  does  not  differ  in  gray  level  from  the 
exterior. 


A  hand  drawn  kidney  bean  shaped  closed  curve  was  used  to 
demonstrate  the  method;  see  Figure  16.  The  curve  was  chain- 
coded  and  curvature  at  each  point  was  measured  by  computing 
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two  slope  vectors  four  chain  links  in  length,  one  on  each 
side  of  the  point,  normalizing  each  by  its  magnitude  and 
taking  the  dot  product  of  the  vectors  [20  ]  •  Using  four- 
neighbor  adjacency,  those  points  adjacent  to  the  edge  which 
were  toward  the  center  of  curvature  were  labeled  "inside", 
and  those  points  adjacent  to  the  edge  but  away  from  the 
center  of  curvature  were  labeled  "outside".  The  probability 
of  belonging  to  each  class  was  computed  by  normalizing  the 
curvature  measure  at  the  adjacent  edge  by  the  maximum  curva¬ 
ture  value  possible.  If  a  point  was  adjacent  to  more  than  one 
edge  point,  supporting  curvature  evidence  (i.e.,  all  adjacent 
edge  points  were  either  convex  or  concave)  increased  class 
probability  in  proportion  to  the  evidence  strength,  while  con¬ 
tradictory  curvature  evidence  (i.e.,  adjacent  edge  points 
having  convex  and  concave  measurements)  created  an  ambiguous 
situation  and  inside/outside  class  probabilities  were  set  equal. 
Points  not  adjacent  to  an  edge  were  also  initially  assigned 
ambiguous,  equal  inside/outside  probabilities.  In  this  experi¬ 
ment,  the  edge  points  are  known  so  no  fuzzy  classification  was 
necessary.  Edge  points  were  assigned  an  edge  probability  of  1 
and  all  other  points  assigned  a  0  edge  probability.  Figure  17a 
shows  the  initial  pixel  classifications  with  the  class  probabi¬ 
lities  rescaled  as  gray  levels  where  g  =  63  -  pI*63  +  .5  = 
pQ*63  +  .5.  Initially  most  points  are  ambiguously  classified 

and  appear  gray.* 

* - 

It  should  be  pointed  out  that  the  high  and  low  initial  values 
at  the  bottom  of  the  concavity  on  the  right  side  seem  to  be 
reversed. 


25 


The  compatibility  coefficients  for  the  relaxation  process 
were  determined  a  priori  based  on  the  constraints  that  inside 
reinforces  inside  and  outside  reinforces  outside;  an  inside/ 
outside  label  pair  is  contradictory,  and  classes  separated 
by  an  edge  do  not  interact.  They  are  displayed  in  Figure  18. 

Figure  17 (b-m)  shows  twelve  iterations  of  relaxation 
applied  to  Figure  17a.  In  this  display  scheme,  as  class  sup¬ 
port  increases,  the  inside  class  goes  to  black,  the  outside 
class  goes  to  white,  and  the  curve  remains  gray.  As  expected, 
the  ambiguity  is  resolved  with  the  interior  of  the  curve 
classified  "inside"  and  the  exterior  classified  "outside". 

The  histograms  of  the  iterations.  Figure  19(a-m),  show  the 
behavior  of  the  process.  Initially,  nearly  all  points  are 
ambiguously  classified  and  show  up  as  a  spike  at  the  middle 
of  the  histogram  (at  value  32).  As  the  process  iterates, 
the  spike  separates  with  the  inside  class  points  migrating  to 
0  and  the  outside  class  points  migrating  to  63.  The  exterior 
points  beyond  the  outwardly  propagating  wavefront  of  "outside" 
support  and  the  curve  points  themselves  remain  ambiguously 
classified.  If  the  process  were  iterated  further,  the  entire 
exterior  would  be  classified  "outside"  and  the  histogram  would 
then  consist  of  three  spikes:  an  "inside"  class  spike  at  0, 
an  "outside"  class  spike  at  63,  and  a  small  curve  spike  at  32. 

Processes  of  this  type  might  be  used  to  model  figure-ground 
ambiguity,  such  as  the  Rubin  vase,  by  initiating  a  propagation 


;♦  'w.+*j** 
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process  at  some 
between  the  two 


particular  sharply  curved  part  of  the  boundary 
regions . 


Detecting  objects  in  a  time  sequence  of  images  is  another 
domain  where  interacting  relaxation  processes  may  be  useful. 
Motion  between  frames  produces  a  disparity  between  the 
images.  The  disparity  contains  basic  relational  information 
about  the  object  which  aids  in  object  definition.  Information 
that  might  be  difficult  to  extract  or  might  even  be  lacking 
in  a  single  image  may  be  available  in  a  pair  of  images. 

Joint  relaxation  processes  can  estimate  the  disparity  or 
velocity  at  each  point  concurrently  with  the  estimation  of 
lightness/darkness  and  edgeness  for  object  detection. 

Presented  here,  for  simplicity,  is  an  example  involving 
a  one-di»uensional  time  varying  sequence  where  velocity  is 
restricted  to  horizontal  motion  at  unit  velocity  (i.e.,  a 
moving  waveform) .  We  give  a  model  for  computing  class  com¬ 
patibilities  for  each  pair  of  adjacent  points  at  each  iteration 
based  on  current  class  estimates  involving  the  interactions 
between  the  light/ dark  process  and  a  motion  direction  process. 
The  model  uses  a  neighborhood  given  by 
ffc_1:  a  b  c 
ffc  :  d  e  f 

ft+l:  g  h  i 

where  ffc  is  the  frame  at  time  t. 


Let  M  =  {N,L,R}  be  the  set  of  motion  direction  estimations 
for  no  motion,  left  motion,  and  right  motion  respectively, 
and  let  G  =  {1,6}  be  the  set  of  light/dark  estimations.  Adja¬ 
cent  class  compatibilities  are  computed  using  the  following 
notation.  Let 


where  a,  b,  a',  b'  are  non-negative  motion  compatibilities 
with  the  following  relative  weight  meanings: 
a  compatible 

b  incompatible 

a'  highly  compatible 

b'  highly  incompatible 

and  I 2»  I3  are  "no  information"  compatibility  matrices. 
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Table 


where  I  denotes  the  "no  information"  coefficients. 

The  principle  behind  these  computations  is  to  base  the 
light/dark  compatibility  coefficients  on  the  motion  estimations 
of  the  point,  and  base  the  motion  compatibility  coefficients 
on  the  gray  level  similarity  of  the  adjacent  points.  The  co¬ 
efficients  calculated  by  these  tables  conform  to  the  constraints 
that  points  moving  with  similar  motion  reinforce  each  other 
if  they  are  both  within  the  object  or  background,  interior 
reinforces  interior,  exterior  reinforces  exterior,  and  motion 
from  the  object  should  not  propagate  to  the  background.  Table 
1  shows  the  intraframe  computation.  Table  2  shows  the  "no 
motion"  situation  where  point  b  in  frame  ffc  ^  becomes  point  e 
in  ffc.  Table  3  shows  the  "right  motion"  case  where  point  a 
in  frame  becomes  point  e  in  ffc.  The  computations  of 

coefficients  for  the  other  adjacent  points  are  based  on  these 
three  tables  with  the  obvious  symmetrical  relations  and  appro¬ 
priate  direction  substitutions. 
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CHAPTER  IV 
CONCLUSION 


We  have  examined  a  fuzzy  classification  scheme  for  blob 
detection  which  bases  its  classification  updating  on  edge  and 
gray  level  information.  We  examined  the  processes  operating 
both  independently  and  together.  We  also  discussed  extensions 
to  these  processes  and  to  the  problem  domain. 

The  edge  relaxation  process  operating  independently  defined 
the  object  after  four  iterations,  but  with  some  edges  being 
brought  out  of  the  background  as  the  process  continued.  How¬ 
ever,  the  results  were  an  improvement  over  the  non-probabilistic 
edge  detector  results.  Interacting  with  the  light/dark  infor¬ 
mation  resulted  in  further  inhibition  of  the  background  edges 
that  did  not  surround  a  blob,  and  yielded  complete  blob  defini¬ 
tion  in  fewer  iterations.  The  borderness  scheme  further  im¬ 
proved  the  results  producing  an  almost  perfect  segmentation. 

In  the  light/dark  relaxation  process  operating  independently 
the  blob  interior  grew  into  the  background.  Interaction  with 
the  edge  information  substantially  contained  the  interior 
growth  and  produced  an  improved  object/background  discrimina¬ 
tion.  In  the  borderness  scheme  the  object  interior  grew  in¬ 
wards  filling  the  object  and  resulting  in  a  further  object/ 
background  discrimination. 


32 


Under  the  premise  that  multi-level  processes  have  faster 
information  propagation  and  are  more  informed  than  single 
level  processes  we  tried  relaxation  on  a  pyramid  structure 
of  successively  resolved  images.  As  this  was  the  first  stage 
of  experimentation,  we  tried  the  default  option  of  computing 
compatibility  coefficients  by  mutual  information.  The  results 
were  no  improvement  over  the  single  level  process.  The  next 
step  appears  to  be  choosing  more  intelligent  coefficients  to 
support  the  object  information. 

Using  curvature  to  provide  an  initial  estimate  of  inside/ 
outside  probabilities  produced  an  accurate  classification  on 
our  experimental  curve.  Now  that  we  have  verified  its  behavior, 
this  information  source  should  be  brought  to  interact  with  the 
edge  and  gray  level  processes.  The  additional  information 
should  produce  a  still  sharper  object/background  discrimination. 

We  also  formulated  a  parallel  model  for  computing  compati¬ 
bility  coefficients  for  blob  detection  in  a  time  sequence  of 
images.  We  restricted  motion  to  its  simplest  case,  horizontal 
unit  motion  in  one  dimension,  to  study  the  object  position/ 
movement  relations  in  successive  frames.  The  model  is  untested 
as  yet  because  of  the  computational  cost  involved.  As  the 
position/movement  relations  become  better  understood,  they  can 
be  generalized  to  less  restricted  domains. 


All  of  the  processes  examined  in  this  paper  are  parallel 
processes.  At  present,  the  time  requirements  for  these  processes 
limit  their  usefulness  as  real-time  object  detectors.  However, 
with  the  development  of  parallel  processors  the  computational 
cost  involved  should  well  be  within  the  requirements  of  real¬ 
time  object  detection. 
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